Papers with large-scale benchmarks
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate factual consistency of text depend on limited data . e.g., generated text can contain factual inconsistencies that are irrelevant to context . |
| Approach: | They propose a new holistic metric that measures factual inconsistencies . they use 4.7M training examples from 7 well-established tasks . |
| Outcome: | The proposed metric outperforms existing metrics on 22 datasets and matches or outperFORMs them. |
PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets with verifiable answers are limited in reliability, diversity, and scalability . a new approach to generate verifikatable data at scale is needed to improve models' performance . |
| Approach: | They propose a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. |
| Outcome: | The proposed framework improves performance on a wide range of puzzles and logic benchmarks. |
SQLAgent: Learning to Explore Before Generating as a Data Engineer (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to large language models fail to generalize in complex, real-world settings due to database-specific nature of SQL reasoning. |
| Approach: | They propose a two-stage LLM-based framework that decouples knowledge acquisition from query generation. |
| Outcome: | The proposed framework significantly improves accuracy over baselines on large-scale benchmarks. |